An Exploration of the Similarities and Differences Between Processing Text- and Visually Based Narratives

The title of this chapter explores to what extent there are shared cognitive systems that support the processing of narratives across print and visual media. An initially obvious answer to the question is no, given that viewing images and reading texts involve different cognitive and brain systems during encoding. In fact, we contend that there are aspects of how narratives are conveyed across media that may lead to differences in encoding that are less obvious. On the other hand, if we assume that mental models provide the basis of comprehension across modalities of experiences (McNamara & Magliano, 2009), surely there are shared cognitive systems that lead to the construction of these mental models (Gernsbacher, 1990; Gernsbacher, Varner, & Faust, 1990; Magliano, Radvansky, & Copeland, 2007). Consider the shot sequence from the James Bond movie Moonraker (Broccoli & Gilbert, 1979), shown in Figure 8.1, which constitutes about 15 seconds of film. Just prior to this sequence, we find Bond on an airplane filled with enemy agents who want to kill him. Bond throws one of the bad guys (who has a parachute) out of the plane and then subsequently gets pushed out of the plane himself (without a parachute). Most viewers of this sequence infer that Bond has the goal to steal the parachute and will be successful in doing so at the start of Shot 6, when Bond starts his dive (Magliano, Dijkstra, & Zwaan, 1996). Upon review of the shot sequence in Figure 8.1, one can see that the sequence is carefully constructed to constrain the possible interpretation of the narrative events. More important, it illustrates some conventions in filmmaking that may not have correlates in text; but at the same time, the sequence also illustrates that there must be overlapping processes between film and text. Consider Shots 1 and 2, which are extreme, high-angle, long shots of two figures in the air and heading toward the earth. These shots establish the nature of a complication or problem for Bond to solve, specifically that he is falling through the air without a parachute. Shot 2 also introduces a potential solution to that problem, namely, the parachute possessed by the bad guy. Although processing the events conveyed in these camera shots may be different in many ways from processing analogous sentences in a text-based version of the narrative, in doing so viewers need to engage in mapping processes across the images that are also central to text